Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a way for sets to zipper with more things #26595

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from

Conversation

lydia-duncan
Copy link
Member

Prior to this work, sets could only zipper with identical other sets (or basically just itself). Zippering with sets of the same length, or with arrays, would result in errors about unequal zippering lengths.

This PR adds a new unstable iterator to Set, contents. This iterator is far from perfect but is a good start at providing this functionality.

Adds a leader/follower pair of iterators to ChapelHashtable to support this effort. These iterators and their support functions allow the hashtable to evenly divide the elements stored in it, instead of evenly dividing the entire hashtable's storage capacity. They make some explicit error calls like normal zippering does to try and keep the error messages in line with the ones generated by the compiler, though I strongly suspect I may have missed some.

While here, I fixed a comment that was missing a closing parenthesis.

Test updates:

  • Updated test/library/standard/Set/these/setZipDifferentSize.chpl to use the new iterator and ensured it worked. This test is the motivation behind making a separate iterator instead of just modifying these, as it demonstrated that we need to be passed the size of the leader or we'd fail to detect zippering size mismatches

New tests:

  • Added a version of setZipDifferentSize.chpl that reversed the order, to show that it still worked
  • Added a test of zippering with an array leader that is the same length as the set
  • Added a test of zippering with an array follower that is the same length as the set
  • Added a test of zippering with an array leader that is shorter than the set
  • Added a test of zippering with an array follower that is shorter than the set
  • Added a test of zippering two sets with different contents (basically, David Longnecker's original test from Zippering of two sets can fail even when they are the same length #15824)
  • Added a test of the serial version of the new iterator, just for completeness
  • Added a test of the standalone version of the new iterator, just for completeness
  • Added a test of zippering an array with the set's toArray result (user code, but not expected to have changed as a result of this effort)
  • Added a test ensuring that we get an unstable warning for all contexts where the new iterator would be called (see [Bug]: Unstable iterator overloads always warn, even if another overload is used #26590)
  • Added a test ensuring that the argument to the iterator matches the size of the set when called in the leader context

Places for improvement:

  • performance is probably terrible, but I wasn't sure how to solve that in the current iterator framework
  • the argument to the iterator is a little weird. Having a default value makes it more reasonable for the serial and standalone versions, but it's still odd to have it be necessary (but without it, I wasn't sure how to avoid a bug when the chunk size and the number of elements interplayed oddly). I also strongly suspect users will not understand what to do with it if they are learning from someone else's code. Hopefully that'll be rare enough that it won't be an issue, though. Or maybe we'll replace it and it'll be fine?
  • I discovered a bug with @unstable while implementing this (see [Bug]: Unstable iterator overloads always warn, even if another overload is used #26590). It'd be nice to resolve that, but not essential for this PR to function.
  • Throwing from non-inlined iterators :)

Passed a full paratest with futures

Adds a pair of leader/follower iterators to ChapelHashtable and a helper method,
and calls them from the Set `these` iterators.

The implementation does frequent recomputation of how to divide up the iteration
space to find full slots.  This can probably be improved on, but I wanted to
take a snapshot of what worked before attempting it.

It also does not help when zippering with arrays yet.  That is also a next step

----
Signed-off-by: Lydia Duncan <[email protected]>
Test comes from issue chapel-lang#15824 and checks zippering two sets of the same length
(but different contents).  Sorts the output so that it is consistent instead of
potentially racy.

----
Signed-off-by: Lydia Duncan <[email protected]>
This is a first step towards being able to zip sets with other types instead of
just other sets

----
Signed-off-by: Lydia Duncan <[email protected]>
This meant always computing the number of chunks and which chunk we were
ourselves, since the array iterator expects the other components of the tuple to
be ranges as well (so we can't use followThis to send that information) and the
argument list for the follower is supposed to match the argument list of the
leader aside from followThis.  This is unfortunate, but given the sizes I
checked, seemed to be okay.

----
Signed-off-by: Lydia Duncan <[email protected]>
Covers:
- converting the set to an array and then zippering
- zippering with the array leading
- zippering with an array when the set leads

With the prior commit, all three work (though toArray already worked prior to
this effort).

----
Signed-off-by: Lydia Duncan <[email protected]>
The original is only failing on one of the two machines I checked, so I'm
curious if this also fails there or if it behaves as expected (it passes on the
machine where the original still works)

----
Signed-off-by: Lydia Duncan <[email protected]>
I'm not sure if this will actually allow us to check for unequal lengths, but
try it and see

----
Signed-off-by: Lydia Duncan <[email protected]>
This argument indicates the size of the leader in the iteration, so that we can
validate if the iteration space is uneven.  It has a default value so that the
non-zippered case is not as annoying to write

Update various tests to use this iterator instead

----
Signed-off-by: Lydia Duncan <[email protected]>
- document new iterator
- remove commented out writelns
- restore old behavior to Set's these iterators, since we're using a different
  iterator
- comment an if statement in _determineEvenChunks
- close a parenthetical in an older comment in ChapelHashtable

----
Signed-off-by: Lydia Duncan <[email protected]>
It'll present a cleaner interface to users

----
Signed-off-by: Lydia Duncan <[email protected]>
Covers:
- Zippering with a shorter array as leader
- Zippering with a shorter array as follower
- Serial version of the new iterator
- Standalone version of the new iterator

----
Signed-off-by: Lydia Duncan <[email protected]>
…f it

Would have added a warning to all the overloads, except for the bug reported in
all the right contexts, whether the bug or the unstable warning gets resolved
first.

----
Signed-off-by: Lydia Duncan <[email protected]>
Use a halt wrapper instead of `__primitive("chpl_error"`, especially because
this is a new error instead of trying to match errors that the compiler can
insert.

While doing that, made the error messages clearer about what was wrong and what
should be done instead (and hopefully head off confusion for the follower case).
And add a test locking in the error message in the leader case

----
Signed-off-by: Lydia Duncan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant